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ABSTRACT 

Limited visibility has been cited as predominant causal factor for both Controlled-Flight-Into-Terrain (CFIT) and 
runway incursion accidents. NASA is conducting research and development of Synthetic Vision Systems (SVS) 
technologies which may potentially mitigate low visibility conditions as a causal factor to these accidents while 
replicating the operational benefits of clear day flight operations, regardless of the actual outside visibility condition. 
Two experimental evaluation studies were performed to determine the efficacy of two concepts: 1) head- worn display 
application of SVS technology to enhance transport aircraft surface operations, and 2) three-dimensional SVS electronic 
flight bag display concept for flight plan preview, mission rehearsal and controller-pilot data link communications 
interface of flight procedures. In the surface operation study, pilots evaluated two display devices and four display 
modes during taxi under unlimited and CAT II visibility conditions. In the mission rehearsal study, pilots flew 
approaches and departures in an operationally-challenged airport environment, including CFIT scenarios. Performance 
using the SVS concepts was compared to traditional baseline displays with paper charts only or EFB information. In 
general, the studies evince the significant situation awareness and enhanced operational capabilities afforded from these 
advanced SVS display concepts. The experimental results and conclusions from these studies are discussed along with 
future directions. 
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1. INTRODUCTION 

The Integrated Intelligent Flight Deck Technologies (IIFDT) project, under NASA’s Aviation Safety Program (AvSP), 
is comprised of a multi-disciplinary research effort to develop flight deck technologies that mitigate operator-, 
automation-, and environment- induced hazards. Towards this objective, IIFDT is developing crew/vehicle interface 
technologies that reduce the risk of pilot error, improve aircraft safety for current and future civilian and military 
aircraft, and proactively overcome aircraft safety barriers that would otherwise constrain the full realization of the next 
generation air transportation system (NGATS). Part of this research effort involves the use of synthetic and enhanced 
vision systems and advanced display media as enabling crew-vehicle interface technologies to meet these safety 
challenges. 

1.1. HUD limitations 

Experiments and flight tests have shown that a Head-Up Display (HUD) can be effectively enhanced with Synthetic 
Vision 1,2,4 . Additionally, NASA LaRC’s Runway Incursion Prevention System (RIPS) 7 and NASA Ames’s Taxiway 
Navigation and Situation Awareness (T-NASA) system 8,9,10,11 conducted several flight and simulation experiments 
evaluating cockpit display technology to enable improved and safer surface operations. Both the RIPS and T-NASA 
concepts used a HUD and a head down display including an electronic moving map (EMM). While great success in 
ground operations was demonstrated with a HUD, the research noted that two of major HUD limitations being in 
ground operations are their monochrome form and a limited, fixed field of regard. A monochromatic display has the 
inherent problem of being unable to use color for information decluttering and information cuing. Coupled with a 
limited field of regard, the display area of a HUD must be carefully designed to provide the pilot with enough 
information without saturating the display with information clutter. 
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1.2. Head-worn displays (HWD) 

Advances in display devices (e.g., electronic flight bags, head-worn devices) have been studied by NASA researchers as 
an alternate and practical method for delivering SVS concepts to the cockpit. HWDs are small, light weight full color 
display devices that can be worn on the head without significant encumbrance (Fig. 1). By coupling the HWD with a 
head tracker, unlimited field-of-regard can be realized. Unlike fixed field of view (FOV) sensors, the camera position 
and orientation for Synthetic Vision can be defined via software; thus, an unlimited field-of-regard is achieved since the 
Synthetic Vision scene is viewable from any virtual camera angle. 

The advantages of the full color, Head Tracked - Head Worn Display (HT-HWD) can directly address the HUD 
limitations shown in RIPS and T-NASA testing. As such, a study was conducted to determine the efficacy of a HT- 
HWD in a taxiing task in a part-task simulator. In addition, the study was used to obtain pilot comments on the concept 
and future enhancements required for using a HT-HWD for surface operations. 



Figure 1 : A Synthetic Vision enhanced HWD with a head tracker has unlimited field of regard. 


1.3. Electronic Flight Bags (EFB) 

Another approach envisioned by NASA is to enhance Electronic Flight Bags (EFB) with Synthetic Vision technology. 
EFBs are being offered by a variety of vendors as a means to eliminate paper in the cockpit and do general computing 
tasks such as airplane performance calculations 12,13 . The first evolution of EFBs consisted of scanning in current paper 
charts. More advanced versions of these EFBs contain a moving ownship symbol or map to enhance the pilot’s 
situational awareness (SA) (e.g., commercially-available Class 2 EFBs present ownship position information on the 
airport surface). The impetus for the introduction of EFBs is the significant reduction in the weight of paper that pilots 
have to carry with them which can weigh upwards of 771bs (e.g., 777-200). Additional applications are continually 
evolving such as the capability to perform performance calculations on the flight deck which has been shown to 
increase gross takeoff weight and substantially reduce engine wear. Therefore, EFBs are likely to increase in 
prevalence and, as a consequence, more and more features and functionality will be introduced. Because of advances in 
computer technology, 3 -dimensional (3-D) graphical EFBs are now possible. Therefore, current prototype concepts are 
being studied by NASA LaRC that would leverage Synthetic Vision System (SVS) technologies to benefit aviation 
safety. 




Figure 2: Paper charts (left), typical Electronic Flight Bag (middle) and the NASA-developed Mission Rehearsal Tool (right). 


Although an experienced pilot is quite familiar the paper format, an electronic version that merely replicates paper 
information is unlikely to significantly increase safety; conversely, studies suggest that electronic charts may be harder 
to read and use 13,14 . Furthermore, while some vendors are pursuing the safety benefits of EFBs (e.g., enhancing surface 
awareness) the tremendous potential of EFBs beyond the more obvious applications (e.g., video surveillance, 
performance calculations, electronic charts and manuals) have yet to be researched particularly with recent advances in 
3-D graphical displays and Synthetic Vision technology. Research at NASA LaRC is focusing on development of EFBs 
to enhancing aviation safety. Figure 2 shows the progression of paper charts to “e-paper” and the NASA-developed 
Mission Rehearsal Tool (MRT), described below. 


1.4. NASA Mission Rehearsal Tool (MRT) 

A significant benefit of presenting terrain information in the cockpit is the potential mitigation, or prevention, of 
controlled- flight- into-terrain (CFIT). Previous NASA research 15 has shown that an egocentric PFD with SVS can 
significantly reduce the potential for CFITs compared to the state-of-the-art available today in terrain alerting and 
warning systems (i.e., Terrain Awareness and Warning System (TAWS) and Vertical Situation Displays). These results 
were confirmed and extended 16 by demonstrating that the addition of an exo-centric navigation display, that provided 
pilots with perspective modes which support motion parallax of terrain information, significantly enhanced situation 
awareness and greatly increased safety margins for various CFIT situations. 

A key finding from those studies was that a display that could allow pilots to rehearse and preview flight path 
information and ATC clearances in relation to 3-D terrain information may significantly enhance the tremendous 
potential of SVS to proactively, in contrast to reactively (i.e., responding to TAWS alerts), prevent flight crews from 
getting into potential CFIT situations. The preview capability would enable flight crews to develop and refine their 
mental model well before any possibility of CFITs could present themselves. Because of the additional safety benefit 
this would provide, NASA has been evaluating such a capability through the use of SVS technology and EFB displays - 
called the “Mission Rehearsal Tool” or MRT. 


The NASA MRT is envisioned as an enhancement to current EFBs by providing an interactive 3-D view. While a 
moving ownship overlaid on electronic charts is available as part of the NASA MRT concept, the planned flight path 
may also be viewed from various orientations to provide improved path and terrain awareness via graphical 2- 
dimensional (2-D) or 3-D perspective display formats. By coupling the path with a terrain database, uncompromising 
terrain awareness relative to the path and ownship is provided. In addition, missed approaches, path deviations, and any 
navigational path can be reviewed and rehearsed before performing the actual task. By rehearsing a particular mission, 






check list items can be reviewed terrain awareness can be highlighted and missed approach procedures can be discussed 
by the flight crew. Further, as Controller Pilot Data Linked Communications (CPDLC) progresses in the industry, MRT 
offers an ideal platform from which data-linked path, flight plan changes, and Air Traffic Control requests can be 
displayed. It is envisioned that the proactive nature of MRT will create an intuitive means of ensuring SA among the 
flight crew in present and future air space environments. 

The primary objective of the NASA research on EFBs is to mitigate currently witnessed problems on the flight deck and 
to prepare for technologies to support emerging operating concepts for the next generation air transportation system, 
such as equivalent visual operations. Today, flight management systems (FMS) transform and display stored FMS data, 
sensors information and programmed/stored flight plans with only ownship information and minimal symbology 
depictions, relegated to airport symbology, NAVAIDs, etc. Pilots currently have to mentally rehearse, memorize, and 
then translate that mental model using a plan view mode of 2-D ND information that presents little or no correlating 
symbology to FAA-approved and airline company charts that specifically define what the flight crew must do for a 
particular procedure. 

MRT attempts to overcome these limitations by blending existing and new technologies together to form a 3-D 
exocentric display. This interactive display creates an immersive display of terrain; current, and predicted flight paths. 
The displays show planned and actual flight paths, obstacles, approach, en route, other chart symbologies, and company 
(and FAA) procedures. The MRT also is designed to allow the pilot to rehearse and preview loaded FMS paths, 
anticipated changes to aircraft routes, new and unfamiliar procedures, etc in an intuitive, 3-D view with full control of 
perspective. By providing a real-time depiction of where the aircraft is in relation to significant terrain on approach and 
departure with preview capability, MRT may provide an enabling technology for safe realization of emerging 4-D 
Required Navigation Performance (RNP) and CPDLC interface functionality. 

1.5. Experimental studies 

Two part-task experimental studies were conducted to determine the efficacy of using HWDs to enhance taxi operations 
and EFBs for enhancing approach and departure operations. 

In the first experiment, full-color HWD display concepts were evaluated in surface operations to address previously 
witnessed display technology limitations. Previous research has shown that a HUD can significantly enhance SA for 
surface operations; however, due to the HUD’s fixed field-of-regard and limited FOV, intuitively portraying turns on 
the HUD can be difficult 10 . Further, information clutter is a driving constraint for the monochromatic HUD. Though 
EFBs are emerging for surface operations, current regulations require the use of paper charts and Class 2 moving map 
EFBs which do not mitigate many issues in today’s complex surface operations. (For an unfamiliar airport, a complex 
airfield can make taxiing a challenging task for the pilot 10 ). 

Second, an additional experimental study was conducted to determine the efficacy of the NASA-developed EFB known 
as the MRT. The purpose of the study was to conduct a proof-of-concept and evaluation of presenting 3-D information 
on an EFB. The objective was to collect data that will enable further development and enhancement of the concept to 
prepare for a high fidelity simulation experiment. Additionally, the study was designed to supplement past research on 
SVS focused on how the technology can mitigate or even eliminate CFITs. Therefore, pilots were exposed to nominal 
and off-nominal situations designed to elicit pilot feedback and usability assessment while also providing a scientific 
methodology to evaluate the potential of the MRT display concept for prevention of CFITs. 

1.5.1. Simulation facility 

Both studies was conducted in the Visual Imaging Simulator for Transport Aircraft Systems (VISTAS) III part-task 
simulator at the NASA LaRC (Fig. 3). VISTAS III is a single pilot fixed-base simulator consisting of a 144 degree by 
30 degree out the window visual, a large field HDD, and pilot input controls. The simulated aircraft in VISTAS III was 
a Boeing 757. 




Figure 3: VISTAS III part-task simulator. 


2. EXPERIMENT ONE: HEAD WORN DISPLAYS 


2.1. Methodology 

For the HT-HWD usability study, two HWD devices were used: 1) 800H x 600V pixel, full color display with optional 
see-through capability; and, 2) a glasses-mounted, full color 640H x 480V pixel, non-see through display. An optical 
head tracker provided the head orientation data. The pilot controls were a tiller, throttles, and differential toe brakes. 
The pilot interacted with the system via a voice recognition system (VRS). The VRS was not a requirement for the 
evaluation of HWD technology but provided an expedient way to allow pilot interaction. A total of six VRS commands 
were used to change the display modes and change display range. 

2.1.1. Evaluation pilots 

Eight pilots participated in the experiment. The pilots consisted of 6 commercial pilots and 2 test pilots. Of the six 
commercial pilots, 2 were captains and 4 were first officers. The 6 commercial pilots had an average of over 13,000 
flight hours. The subjects were given a 30-minute briefing to explain the display concepts and the evaluation tasks. 
After the briefing, a 1 hour training session was conducted to familiarize the subjects with the VISTAS III simulator, the 
HWD devices, and the piloting task. Following training, 2.5 hours of data collection was conducted. The total time for 
a subject was approximately 4 hours. 

2.1.2. HWD evaluation task 

Pilots conducted taxi operations at Reno, NV (FAA identifier: RNO) based on simulated data-link taxi clearances. The 
HWD device, the display concept, and weather were varied. Pilots were instructed to taxi at a speed they thought 
appropriate for the task. Two taxi routes were used in the study: 1) a runway to gate route, and 2) a gate to runway 
route. The runway to gate taxi route was to exit Runway 16R via November taxiway, turn onto Bravo taxiway, and turn 
on the Golf taxiway proceeding to the gate. The gate to runway route was to taxi from the Mercury Aviation Center 
apron onto the Charlie taxiway, cross Runway 25 to Papa, and hold short of Runway 34L. The weather state for the out 
the window scene was varied between clear day with unlimited visibility and fog, 1000 foot runway visibility range 
(RVR). 

2.1.3. HWD conditions 

Three different display conditions were used: 1) a paper chart, 2) an advanced EFB type display consisting of a 2-D 
track up moving map with the cleared taxi route and ownship symbol and 3) HWD concepts. 


For the HWD concepts, two HWD devices were tested. The first HWD was nominally see-through with 800x600 pixel 
resolution. The display could also be made non-see-through by closing an opaque door on the display. The FOV was 
approximately 23° horizontal by 16.5° vertical. The second HWD had 640x480 pixel resolution with an approximate 
FOV of 14° horizontal by 10.4° vertical and was non-see-through. 

For each HWD, there were two display concepts tested (Fig. 4): 1) Single Mode and 2) Multi-Mode. Single Mode 
consisted of a 2-D moving map, plan view display. This concept does not require head tracking. The Multi-Mode 
display allowed the pilot to choose between 4 different display modes: 1) a text display of the taxi clearance, 2) a 2-D 
moving map (the Single-Mode presentation), 3) a zoomed-in 2-D moving map for precision surface guidance and 4) a 
3-D perspective display. 

The 3-D perspective display was evaluated with and without head-tracking. For the head-tracked condition, the FOV 
presented on the display corresponded with that of the display device (i.e., it was conformal). For the non-head-tracked 
case, two FOVs were tested: 1) the FOV of the display device (i.e., the same FOV as conformal, but non-head-tracked); 
and, 2) a fixed field-of-regard approximately the same as a HUD FOV (30° horizontal by 24° vertical). 
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Figure 4: The four display modes for the Multi-Mode surface operations concept. 
In the Single Mode case, only the 2-D moving map concept was displayed to the pilot. 


The usability evaluation was conducted in 4 blocks. For each of the four blocks, the weather (clear or 1000 foot RVR) 
and display mode (Multi-Mode and Single Mode) were held constant. For Block 1, the outside weather was clear and 
unlimited visibility and the display mode was pilot-selectable (Multi-Mode). Block 2 was that same as Block 1, 
however, the outside weather was 1000 foot RVR. The taxi clearance for all of the data runs in Blocks 1 & 2 was the 
runway-to-gate route described above. For Blocks 3 and 4, the display concept was held constant to the Single Mode 
presentation. For Block 3, the weather was clear and unlimited visibility; while for Block 4, the outside weather was set 
to 1000 foot RVR. In Single Mode, there was neither head tracking nor any FOV variations; thus, the only variation 
was the HWD display device (800x600 pixel open, closed and the 640x480 pixel display). The taxi clearance for all 
data runs in Blocks 3 & 4 was the gate-to-runway route described above. 


2.2. HWD results 

The study was used to confirm the efficacy and acceptance of using a HWD for surface operations. As such, the results 
of the HWD usability study were principally subjective. The pilot comments were sought to improve the concepts and 
help focus more rigorous follow-on research. Figure 5 shows the twelve HWD variations tested. 







Figure 5: Display conditions for the HWD usability study. 


2.2.1. Multi-Mode display concept with clear weather (Block 1) 

Using the Friedman test, the rank ordering of the display types by the evaluation subjects was significant. The highest 
ranked display type was the 800x600 pixel display (open) with a conformal FOV for the 3-D perspective mode and the 
head tracker on. The lowest ranked display type was the non-head-tracked 800x600 pixel display (closed) with the 
conformal FOV for the 3-D perspective display mode. 

2.2.2. Multi-Mode display concept with 1000 foot RVR weather (Block 2) 

Using the Friedman test, rank ordering of the display types by the evaluation subjects was again significant. The 
highest ranked display type was the head-tracked 800x600 pixel display (open) with the conformal FOV. The lowest 
rank display type was the non-head- tracked 800x600 pixel display (closed) with the conformal FOV for the 3-D 
perspective display mode. 

2.2.3. Single Mode display (Blocks 3 & 4) 

Regardless of the visibility condition, pilots ranked the 800x600 pixel display (open) and the 640x480 pixel display 
higher than the 800x600 pixel display (closed). For unlimited visibility, the rank order was significant. The 800x600 
pixel display (open) and the 640x480 pixel display had the same highest ranking of 1.62 and the 800x600 pixel display 
(closed) had the lowest ranking of 2.75. For 1000 foot RVR visibility, the rank order was again significant with the 






800x600 pixel display (open) and the 640x480 pixel display having the same highest ranking of 1.5 and the 800x600 
pixel display (closed) having the lowest ranking of 3.0. 

2.2.4. SA-SWORD 

Analysis, using SA-SWORD post- test, showed statistically significant differences across all display types for subjective 
Situation Awareness (SA). There were 3 subsets based on Student-Newman-Keuls (SNK) at a = 0.05. In this paired- 
comparison test, the non-head-tracked display comparison was collapsed. The HUD FOV and Conformal FOV display 
concepts were treated as one, with the EP assuming the optimal or pilot-preferred FOV. 

The 800x600 pixel display (closed), Single Mode had significantly less SA than all other display types tested. 
Conversely, the 800x600 pixel display (open), Multi-Mode with the conformal FOV for the 3-D perspective display had 
significantly higher SA than the following display types: 1) 800x600 pixel display (closed), non-conformal FOV, Single 
Mode; 2) 800x600 pixel display (closed), non-conformal FOV, Multi-Mode; 3) 640x480 pixel display, non-conformal 
FOV, Multi-Mode; 4) 640x480 pixel display, non-conformal, Single Mode but no significant differences with the 
following display types: 1) 800x600 pixel display (open), non-conformal, Single Mode; 2) 640x480 pixel display, 
conformal, Multi-Mode; 3) 800x600 pixel display (closed), conformal, Multi-Mode; and 4) 800x600 pixel display 
(open), non conformal, Multi-Mode. 

A post-test paired comparison questionnaire was conducted in terms of runway taxi/surface situation awareness using 
the displays tested (pilot-preferred Multi-Mode HWD and pilot-preferred Single Mode HWD) and current/near-term 
airline equipage (paper charts and EFB moving map). The results showed that there were statistically significant SA 
differences depending upon display media type. Post-hoc tests show three unique subsets. The pilots’ preferred HWD 
concept was Multi-Mode as it gave significantly greater surface SA than the three other types (preferred Single Mode, 
EFB moving map, paper charts). Pilots’ ranked the Single Mode as having significantly higher SA than EFB moving 
map and paper charts. There were no appreciable differences in SA between EFB taxi map and paper charts. 

2.2.5. Workload 

Subject pilots were given a paired comparison questionnaire regarding workload during runway taxi/surface operations 
using the displays tested (pilot-preferred Multi-Mode HWD and pilot-preferred Single Mode HWD) and current/near- 
term airline equipage (paper charts and EFB taxi-map). The results showed that there were statistically significant 
workload differences depending upon display media type. Post-hoc tests show two unique subsets. Pilots rated paper 
charts as having significantly greater workload than the other three display media types (pilots’ preferred Multi-Mode, 
pilots’ preferred Single Mode, and EFB with moving map). There were no appreciable differences in workload among 
the pilots’ preferred Multi-Mode, pilots’ preferred Single Mode, and EFB moving map display media types. 

2.2.6. Display media 

At the end of the test, pilots were asked to rank-order the various display concepts they had seen by preference. 
Analysis of the data showed that display media type rankings were significant. The pilots’ display media ranking order 
was Multi-Mode (1.1), Single Mode (1.9), EFB (3.0), and Paper (4.0). For the Multi-Mode concept, pilots were asked 
to rank the 4 modes (Fig. 4). Analysis of the results showed that the Multi-Mode concept rankings were significant. 
The rank-order results were the 2-D moving map mode (1.8), the 2-D precision guidance mode (2.0), the 3-D 
perspective mode (2.4), and the clearance text mode (3.9). 

2.3. HWD experiment discussion 

Experiment One was designed to demonstrate the efficacy of a head-worn display which provides unlimited field-of- 
regard Synthetic Vision for surface operations. The results demonstrate that providing pilots with the ability to virtually 
see well beyond visual range can significantly increase situation awareness and task performance on the airport surface. 
Pilots were better able to perform the taxiing evaluation task and reported significantly higher situational awareness 
with the HMD concepts compared to an electronic moving map or paper charts of the airport environment. 
Furthermore, the study provided tremendous insight into future design and development of head-worn displays, 
including hardware considerations and methods for integration of display modes. 



2.3.1. HWD hardware considerations 


Two significant hardware considerations were evinced from the HWD experiment. Nearly all pilots rated the 800x600 
pixel display higher because it had higher resolution than the 640x480 display and it was see-through. The higher 
resolution improves the readability of the display especially for text and numbers. This finding is consistent with past 
research findings that higher resolution displays are generally preferred by pilots. Additionally, pilots preferred not to 
have their forward vision blocked even by the small 640x480 pixel display. The see-through capability allowed pilots 
to continue their nominal out the window surveillance of the airport environment during taxi. Also, the see through 
display provided pilots with confidence that the display was aligned with the scene. 

2.3.2. HWD mode integration 

With regard to display mode integration, pilots reported that the four modes (clearance, 2-D precision guidance, 2-D 
moving map, and 3-D perspective) each have relative merits in supporting taxi operations. For complex operational 
environments, the data-link textual clearance mode was reported to be of significant value in ensuring compliance with 
ground instructions. However, this mode would likely not by itself be enough to improve aviation safety. It is only 
when combined with the other modes that its potential becomes evident. In fact, this observation was witnessed for 
each of the modes. For example, pilots reported that the 2-D moving map was of substantial benefit but it did not 
provide them with the precision needed from the 2-D precision guidance. However, the 2-D precision guidance only 
provided local guidance and they lacked global situation awareness. The 3-D perspective helped give pilots a sense of 
immersion; that is, the feeling that they were looking outside the cockpit into the real-world. This resulted in high 
situation awareness but again, pilots felt that the precision and the “big picture” were missing. Therefore, each of the 
modes contributed something unique, and the display concept was limited in its efficacy only because pilots did not like 
to have to continuously switch between modes to extract the necessary information that each mode provided separately. 
As a consequence, pilots unanimously stated their preference for a more integrated display that would blend the various 
display modes together to reduce the workload and improve accessibility to display information. 

These results have highlighted several technology trends which are now being researched to further and fully develop a 
unified, unlimited field-of-regard Synthetic Vision HWD. Many challenges remain, but the potential to significantly 
improve surface operations using HWD concepts seems clear from these data. 

3. EXPERIMENT TWO: ELECTRONIC FLIGHT BAG 


3.1. Methodology 

3.1.1. Simulation facility 

The MRT display was mounted in the VISTAS III simulator (Fig. 3). MRT was rendered on a tablet PC with a 1.6 GHz 
Pentium M processor, 1 GB of memory and Microsoft Window XP Tablet edition operating system. The MRT 
software was developed in-house by NASA. Interaction with MRT was via a stylus pen. The HDD consisted of an 
ARINC Size D (7 square inch display surface) PFD and ND. The PFD was a typical electronic attitude display 
indicator (EADI) presentation with a velocity vector and guidance cue (the “ball”). The ND was enhanced with Terrain 
Awareness and Warning System (TAWS) and a Vertical Situation Display (VSD) which provided graphical terrain 
depiction and alerts (Fig. 6). 




Figure 6: Baseline PFD and ND displays used for MRT evaluations. 


3.1.2. Evaluation pilots 

Eight commercial “glass cockpit” pilots participated in the MRT usability study. The 8 pilots consisted of five captains 
and three first officers flying for major airlines and had significant line operational experience (> 8000 hours). The 
subjects were given a 30-minute briefing to explain the overall display concepts, background on EFBs, evaluation tasks, 
and relevant procedural information necessary to conduct the experimental session. After the briefing, pilots were 
presented with an informed consent form followed by a E5 hour training session to familiarize the evaluation pilots 
with the VISTAS III simulator, the MRT display, the e-paper display, and expected piloting tasks. Following training, 
2 hours of data collection was conducted. The total time for a subject was approximately 4 hours. 

3.1.3. Evaluation tasks 

Pilots flew two departures (KREMM, Cottonwood-2) and three approaches (FMS-B, LDA25, Circle-To-Land to 
Runway 07) at Eagle-Vail Regional County Airport (FAA Identifier: EGE). During evaluation runs, pilots were given 
simulated audible ATC calls which were also presented as controller-pilot data link communication messages on the 
MRT display condition. Pilots were instructed that the main focus of the study was to evaluate the MRT display and 
not path performance. For all runs except two visual arrivals, the weather conditions were fog with 2000 foot visibility 
and light snow falling at the rate of 75 mm/hour. The visual approach was conducted at a simulated late afternoon time 
with light snow falling. 

3.1.4. CFIT evaluation tasks 

Unknown to the pilots, there were also two CFIT trials presented: an FMS-B approach with an amended clearance that 
directed the aircraft into terrain and a KREMM departure with incorrect guidance (Fig. 7). The FMS-B scenario began 
at 13,100 MSL with an ATC clearance to conduct the nominal approach procedure to Runway 25. Before reaching the 
initial approach fix of MCCOY, the pilot was given an amended clearance to fly direct to BEVEY. The clearance also 
included instructions to descend to and maintain 1 1,000 MSL. This course put the aircraft below terrain with peaks at 
11,653 MSL. The CFIT KREMM departure from Runway 07 was set-up by commanding a 052° heading off of the 
SXW VOR instead of the nominal 059° radial which resulted in a course that came within 100 feet AGL of terrain. In 
addition, the engine-out circling procedure was not done in either training or evaluation trials even though single engine 
power was simulated. Pilots saw each CFIT scenario when flying with either the MRT or e-paper display condition as 
randomly assigned across pilots. For all display conditions, TAWS and VSD presented terrain alerts to the pilots. 



Figure 7: Nominal and CFIT KREMM departure (left) and FMS-B en-route (right) evaluation tasks. 


3.1.5. Evaluation display conditions 

Three display conditions were used in the study: 1) paper charts, 2) e-paper which consisted of an electronically 

scanned chart with ownship symbol showing current position, and 3) the MRT. The paper charts condition represented 
a baseline for comparison and consisted of published approach and departure procedures and specialized major airline 
company procedural technique guides and charts. The e-paper condition consisted of an electronic scan of the charts 
used in the paper charts condition with the addition of a moving ownship symbol (left image in Fig. 8). Finally, the 
MRT display (Figure 8-10) consisted of the e-paper display concepts (as a pilot-selectable mode option) but also 
presented a 2-D plan view of the 3-D terrain database. The 2-D plan view mode is analogous to a ND with a terrain 
database displayed. In addition, the MRT display contained a number of EFB enhancements which were described in 
Section 1.4. For instance, in Figure 8, the middle image of the MRT shows a preview of a data-linked path that goes 
into the terrain (i.e., the pilot should reject the clearance). Blue dots denote checklist item changes and red spheres are a 
visual alert that the path is less than 1000 feet AGL. The CPDLC capability of MRT display is shown in Figure 9. 



Figure 8: E-paper (left) and the MRT display (middle & right) concept. 
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Figure 9: CPDLC capability of MRT display. 
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Figure 10: Techniques and procedures capability of MRT display. 





3.1.6. Experiment design & procedure 

The experiment matrix was a randomly assigned 3 display conditions (paper, e-paper, and MRT) by 5 tasks (LDA25, 
FMS-B, Circle-To-Land to runway 07, KREMM departure, Cottonwood-2 departure) by 2 CFIT tasks (en route, 
departure) mixed- subjects partially factorial design. All pilots flew the Localizer DME (LDA25) approach to Runway 
25 with all three display conditions. The use of paper charts was to provide a baseline rating for comparison of e-paper 
and MRT to currently available technology onboard Part 121 aircraft today (i.e., paper charts, electronic charts without 
ownship). The remaining nominal evaluation tasks (Cottonwood-2 departure, Circle-To-Land to Runway 07) were 
flown with both e-paper and MRT display concepts. During training, pilots flew all five evaluation tasks, including the 
FMS-B approach and KREMM departure, with all display concepts to familiarize and prepare them for the data 
collection trials. The pilots were not exposed to any CFIT events during training, thus, the FMS-B did not have the 
amended clearance causing a CFIT and the KREMM departure was a nominal departure. 

The experimental trials consisted of three LDA25 approaches (paper, e-paper, MRT), two Circle-To-Land approaches to 
Runway 07 (e-paper, MRT), two Cottonwood-2 departures (e-paper, MRT), an en route CFIT (FMS-B), and a departure 
CFIT (KREMM) for a total of 9 experimental trials. During the experimental trials, the FMS-B approach and KREMM 
departure tasks were assigned as a between-subjects factor and pilots were randomly assigned to fly these tasks with 
either the e-paper or MRT display concepts. These evaluation tasks were designed to present a potential CFIT situation 
as described in the section above. 

3.2. NASA MRT results 

All dependent variables were analyzed using multivariate analyses of variance (MANOVA) and subsequent univariate 
ANOVAs based on significant Pillai’s Trace results. Significant main effects were subjected to post-hoc pair-wise least 
significant difference tests. An alpha (a) level of 0.05 was used as the criterion for determining significant effects of 
results. All data was analyzed using Mauchly’s Test of Sphericity and appropriate statistics were used (e.g., 
Greenhouse-Geisser) when the assumption of sphericity was violated. 

No significant effects were found for any dependent variable for evaluation task (p > .05) or the interaction of 
display*task (p >.05). Only main effects for display condition are consequently presented. Data was collected for all 
three display conditions for LDA25 evaluation task only to allow baseline comparisons to e-paper and MRT display 
conditions. The pattern of results between e-paper and MRT displays for the other departure and approach evaluation 
tasks duplicate those found with the LDA25 evaluation task. Therefore, the text describes the findings of data collected 
during the LDA25 approach to minimize space devoted to similar results reporting. Table 1 presents the results for each 
dependent variable for all three display condition for LDA25 approach task. However, Table 2 presents the combined 
results for the remaining departure and approach tasks for e-paper and MRT display conditions. 

3.2.1. Situation Awareness 

An Analysis of Variance (ANOVA) showed a significant effect for display condition, F(2,14) = 12.676, p < .001. Pilots 
rated the MRT display (M = 6.250) significantly higher in situation awareness rating technique 17 (SART = 
understanding - {resources - demand}) compared to either e-paper (p = .048; M = 4.625) or paper charts (p = .002; M = 
3.125) which were also significantly different from each other (p = .020) based on the post-hoc pair-wise least 
significant difference test. 



Table 1 : Statistical results for LDA25 evaluation task comparing all three display concepts. 



MRT Display 

e-paper 

Paper charts 

Significance 

Partial 

Eta- 

Squared 

Observed Power 








SART 

6.250 

4.625 

3.125 

.001 

.644 

.986 

Workload 

2.875 

2.875 

3.500 

.07 

.313 

.514 

Task Awareness 

5.875 

5.125 

4.250 

.001 

.611 

.971 

Terrain Awareness 

6.375 

3.750 

2.875 

.0001 

.926 

1.00 

Ownship 

Awareness 

6.250 

4.625 

3.750 

.0001 

.788 

1.00 

Path Awareness 

6.250 

4.500 

3.625 

.0001 

.739 

.999 

ATC Awareness 

6.250 

5.125 

5.000 

.0001 

.711 

.998 

Ease of Use 

5.875 

5.125 

4.375 

.01 

.482 

.832 


Table 2: Statistical results for approach and departure tasks (except LDA 25) comparing e-paper and MRT. 



MRT Display 

e-paper 

F- Value 
(df= 1,7) 

Significance 

Partial 

Eta- 

Squared 

Observed Power 








SART 

4.458 

4.250 

0.228 

.647 

.032 

.070 

Workload 

3.00 

3.292 

1.874 

.213 

.211 

.221 

Task Awareness 

5.792 

5.167 

13.235 

.008 

.654 

.875 

Terrain Awareness 

6.542 

4.250 

115.710 

.0001 

.943 

1.00 

Ownship 

Awareness 

6.375 

4.792 

35.592 

.001 

.836 

.999 

Path Awareness 

6.417 

4.667 

32.495 

.001 

.823 

.998 

ATC Awareness 

6.208 

5.125 

21.509 

.001 

.754 

.976 

Ease of Use 

5.958 

5.292 

3.613 

.099 

.340 

.376 


3.2.2. Terrain awareness 

ANOVA results evinced that pilots rated terrain awareness significantly higher for the MRT display (M = 6.375) than 
either the e-paper (p = .0001; M = 3.750) or paper chart (p = .0001; M = 2.875) display conditions, F(2,14) = 87.431, p< 
.0001 which were also significantly different from each other (p = .021) based on the post-hoc pair-wise least significant 
difference test. 


3.2.3. Workload 

An ANOVA revealed a non-significant trend for the overall main effect for the modified workload estimation scale 
ratings for mental workload, F(2,14) = 3.182, p> .07. Pair-wise comparisons between the conditions did reveal 
significant effects between e-paper (M = 2.875) and paper chart (M = 3.50) based on estimated marginal means (mean 
difference = .625; std. error = .183, p = .01 1). No differences were found between MRT (M = 2.875) and the other two 
display conditions. 

3.2.4. Task awareness 

There was a significant main effect for display condition for task awareness, F(2,14) = 10.975, p < .001. Post-hoc least 
significant difference test revealed that pilots rated the MRT display (M = 5.875) significantly higher for task awareness 
than either e-paper (p = .020; M = 5.125) or paper charts (p = .006; M =4.250), and that e-paper was rated significantly 
higher than paper charts (p = .041). 





3.2.5. Ownship position and path awareness 

There were significant main effects found for display condition for both ownship position estimation, F(2,14) = 26.060, 
P <.0001 and path awareness, F(2,14) = 19.863, p<.0001. Overall, post-hoc least significant difference tests showed 
that pilots provided higher Likert scale ratings for MRT for ownship position (M = 6.250) than for either e-paper (p = 
.001; M = 4.625) or paper chart (p = .001; M = 3.750) which were significantly different from each other (p = .021). 
Similarly, pilot rated the MRT higher (M = 6.250) for path awareness compared to either e-paper (p = .004; M = 4.50) 
or paper charts (p = .002; M = 3.625) which were also significantly different from each other (p = .021). 

3.2.6. ATC Communications Awareness 

The results for awareness of ATC communications show a significant main effect between display conditions, F(2,14) = 
17.216, p < .0001. Post-hoc least significant difference tests reveal that pilots rated the MRT display significant higher 
in ATC communication awareness (M = 6.250) than either the e-paper (p = .001; M = 5.125) or paper chart (p = .001; 
M = 5.00) display conditions. There were no significant differences found between e-paper and paper chart display 
conditions (p = .598). 

3.2.7. Ease of Use 

An ANOVA found a significant main effect for ease of use for display condition, F(2,14) = 6.517, p< .01. Pilots rated 
the MRT display conditions (M = 5.875) to be significantly easier to use than the paper charts (p = .026; M = 4.375). 
There was also a significant effect found between e-paper (M = 5.125) and paper charts (p = .048). However, there were 
no differences found between the MRT display condition and e-paper (p = .080). 

3.2.8. CFIT scenario results 

There were significant differences in how pilots responded to the CFIT situations dependent upon which display 
condition they were using during the CFIT event. Regardless of CFIT situation, all pilots recognized the potential CFIT 
situation and none of the eight pilots actually hit terrain. For the e-paper display condition, all four pilots accepted the 
ATC-amended clearance during the en route CFIT scenario and did not recognize the impending CFIT until terrain 
alerts appeared on the TAWS and VSD. On average, pilots responded to the CFIT with a vertical “pull-up” maneuver 
approximately 70 seconds before terrain impact and avoided terrain an average of 600 ft. In two cases, pilots missed the 
terrain by 128 ft and 223 ft. In contrast, with the MRT display, pilots rejected the ATC clearance because the preview 
capability clearly showed the amended clearance directed them toward hazardous terrain. Therefore, pilots NEVER 
were put in a situation in which they had to initiate a terrain avoidance maneuver unlike the pilots flying with the e- 
paper display. The same was true for the KREMM departure CFIT situation. All four pilots with the MRT display 
immediately recognized the erroneous flight path and did not take-off in contrast to all four pilots with the e-paper that 
recognized the impending CFIT only after departing and receiving TAWS and VSD alerts. With the e-paper, pilots 
recognized the CFIT situation approximately 2 nm from terrain impact and avoided terrain by 300 ft or less. 

To quantify this safety benefit, the location of average detection of potential CFIT for both display conditions are shown 
in Figure 1 1 during the en route and departure scenarios, respectively. The yellow dots (also labeled Synthetic Vision) 
in Figure 11 represent the average CFIT detection with the MRT concept and the red dots (also labeled E-paper) 
represent the average CFIT detection for the e-paper concept. In Figure 12, the left image shows the en-route CFIT 
scenario in the e-paper condition; the right-hand figure shows the MRT display for comparison (though the pilot did not 
have the MRT for this trial). In Figures 13 and 14, the head-down displays and the e-paper concept, respectively, are 
shown during en-route CFIT situation. The right-hand picture in Figure 14, shows the MRT presentation that the pilots 
would have seen in both real-time and rehearsal/ preview modes, if they had accepted the same flight plan. 

After each CFIT trial, pilots were asked to fill out a questionnaire asking specific questions on a Likert Scale (0 to 7) 
regarding the CFIT scenario they had just experienced. An ANOVA showed that pilots rated their situation awareness 
significantly higher with the MRT (M = 7.00) than with the e-paper display (M = 3.875), F(l, 12) = 69.44, p < .0001. 
Pilots also rated the amount of time they had to respond and their response to the CFIT situation was significantly faster 
and more proactive (“completely aware and proactive to avoid need for evasive response”) with the MRT (M= 6.625) 
compared to e-paper (M = 3.375; “Aware but reactive with barely sufficient time to take evasive response”). For those 
pilots that had the MRT display, they stated that the flight path and CPDLC rehearsal and preview capability added 



100% to their situation awareness and significantly enhanced their ability to detect and recognize a potential CFIT 
situation. In Figure 15, the MRT vividly depicts why the pilots, who had the MRT concept, rejected the ATC amended 
clearance during the en-route CFIT scenario and immediately recognized the erroneous path during the departure CFIT 
scenario, respectively. Similarly, in Figure 16, a snap-shot of the MRT preview of the KREMM departure taken before 
the run began shows the clear and unambiguous nature of the MRT display for CFIT prevention. 




Figure 11: Location of average detection of en-route (right) and departure (left) CFIT. 



Figure 12: En-route CFIT example with e-paper (left) and MRT (right). 




Figure 13: PFD and ND depicting departure CFIT situation with e-paper. 
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Figure 14: En-route CFIT example with e-paper (left) and MRT (right). 




Figure 15: En-route MRT display upon receiving amended clearance (white dashed lines) and in preview (right). 



Figure 16: MRT display during pilot preview of nominal (left) and CFIT KREMM departure path (right). 


3.3. Electronic Flight Bag experiment discussion 

The purpose of Experiment Two was the evaluation of a synthetic vision electronic flight bag display concept compared 
to existing Class III EFBs and paper charts. The focus was on key features that could enhance the potential aviation 
safety benefits of EFBs. Toward that end, the MRT display was designed to introduce intuitive terrain awareness and 





provide needed functionality targeted at specific safety concerns and operational concepts - both for today and in 
support of future operating concepts for the next generation air transportation system. The functionality included 
CPDLC interface technology; graphical depiction of flight path and terrain; preview/rehearsal; flight techniques and 
procedural memory aids; and 3-D graphical charts. 

3.3.1. Enhancement of Electronic Flight Bags 

The results of Experiment Two evinced that the MRT is a substantial enhancement over existing electronic flight bags 
by providing pilots with added functionality not currently present with today’s EFBs. Pilots reported significantly 
greater situation awareness for terrain, ownship position and flight path, task and procedures, and ATC communications 
without an increase in mental workload. Although “e-paper” with moving ownship was reported to greatly increase 
aircraft positional awareness compared to flying with only paper charts, the “e-paper” concept only marginally 
enhanced overall situation awareness. Instead, pilots felt that the addition of CPDLC capability combined with 3-D 
graphical depiction of ATC clearances, terrain, flight path, and aircraft position together with the rehearsal/preview and 
technique/procedural memory aids distinctly provided advantages over the e-paper concept alone. 

3.3.2. Prevention of Controlled-Flight-Into-Terrain 

The finding that MRT was a significant enhancement of electronic flight bags was most evident in the CFIT results. 
Each pilot was exposed to two potential CFIT situations during the experiment: an en-route and departure CFIT. The 
en-route CFIT consisted of an erroneous ATC clearance that put the aircraft on a heading and altitude trajectory that 
collided with terrain. While all pilots with the e-paper recognized the impending CFIT before impact, the pilots had to 
perform aggressive vertical maneuvers and all reported that they had little time to take evasive response; in other words, 
the pilots were reactive to the situation and rated their situation awareness low despite having an electronic 
topographical map showing terrain higher than their cleared altitude. In contrast, pilots experiencing the potential en 
route CFIT with the MRT had significantly higher situation awareness because they were able to preview the CPDLC 
clearance and see three-dimensionally how flight path intersected the terrain; that is, they were proactive in their 
response. The pilots were able to easily recognize the potential CFIT and reply “unable” to the ATC clearance because 
of the terrain conflict. This was true as well of the KREMM departure potential CFIT which involved an erroneous 
departure guidance path. Pilots with the e-paper narrowly avoided the terrain because they unaware of the terrain until 
the situation showed itself to be a danger through TAWS and VSD alerts. These reactive alerts only provided the pilots 
with approximately 60 seconds before impact with terrain to perform an evasive maneuver. The only maneuver 
available is a pull-up which resulted, on average, in terrain clearance by less than 300 ft AGL. On the other hand, pilots 
with the MRT recognized the erroneous flight path during preview of the departure path and, consequently, had time to 
correct the path before the aircraft even took-off. Together, the results show again the difference in advantage afforded 
to synthetic vision systems compared to baseline displays. The synthetic vision system MRT EFB provides proactive, 
rather than reactive, alerting and awareness for pilots to potential terrain hazards, CFIT avoidance, and flight plan 
changes. 


4. CONCLUSIONS 

The experiments revealed numerous future directions to better optimize and develop these concepts. One future 
direction will involved the integration of an enhanced vision sensor technology with the optimized HWD concept 
(based on the findings of Experiment One). More complex surface operations and geometries (such as at Chicago 
O’Hare International Airport) will be used and compared against existing technologies (such as T-NASA display 
system, which incorporates HUD surface ops symbology and a moving map display of information). Another future 
direction will involve the further enhancement of the MRT display and evaluate a more mature concept in high fidelity, 
two-crew simulation experiment for complex operational and terrain-challenged international airline operations. 
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